Papers with real-world environments
Seeded self-play for language learning (D19-64)
Copied to clipboard
| Challenge: | Current methods for learning human language are too data inefficient to learn it in this way. |
| Approach: | They propose to train a meta-learning agent in simulation to interact with populations of pre-trained agents, each with their own distinct communication protocol. |
| Outcome: | The proposed algorithm minimizes the number of on-policy interactions while learning human language while minimizing the number on-political interactions. |
NavA3: Understanding Any Instruction, Navigating Anywhere, Finding Anything (2026.acl-long)
Copied to clipboard
Lingfeng Zhang, Xiaoshuai Hao, Yingbo Tang, Haoxiang Fu, Xinyu Zheng, Pengwei Wang, Zhongyuan Wang, Wenbo Ding, Shanghang Zhang
| Challenge: | Existing embodied navigation methods struggle with such tasks due to their limitations in comprehending high-level human instructions and localizing objects with an open vocabulary. |
| Approach: | They propose a hierarchical framework for long-horizon navigation that integrates human instructions with 3D scene views. |
| Outcome: | The proposed model achieves SOTA results and can complete long-horizon navigation tasks across different robot embodiments in real-world environments. |
EgoSpeak: Learning When to Speak for Egocentric Conversational Agents in the Wild (2025.findings-naacl)
Copied to clipboard
Junhyeok Kim, Min Soo Kim, Jiwan Chung, Jungbin Cho, Jisoo Kim, Sungwoong Kim, Gyeongbo Sim, Youngjae Yu
| Challenge: | EgoSpeak predicts when an agent should begin speaking based on egocentric streaming video. |
| Approach: | They propose a framework for real-time speech initiation prediction in egocentric streaming video by modeling the conversation from the camera wearer's first-person perspective. |
| Outcome: | The proposed framework outperforms random and silence-based baselines in real time and highlights the importance of multimodal input and context length in effectively deciding when to speak. |
Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Reinforcement learning (RL) has re-emerged as a natural approach for training interactive LLM agents in real-world environments. |
| Approach: | They propose a variant that operates on a turn-level MDP formulation, instead of the commonly used token-level one. |
| Outcome: | The proposed method is more robust than the widely used GRPO algorithm and more efficient than token-level MDPs. |
AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing agents lack generalization and specialization capabilities for open-ended tasks . specialized generalists are often underdeveloped in real-world environments . |
| Approach: | They propose a platform to dynamically integrate heterogeneous agents for automating computer tasks . they propose specialized generalist agent MetaAgent with the AgentToken strategy . |
| Outcome: | The proposed platform expands capabilities of existing agents in generalization and specialization . it can be used to automate open-ended tasks in real-world environments . |
A Picture is Worth a Thousand Words: Language Models Plan from Pixels (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work uses pre-trained language models to reason about plans from text instructions in embodied visual environments. |
| Approach: | They propose to use pre-trained language models to reason about plan sequences from text instructions in embodied visual environments. |
| Outcome: | The proposed approach outperforms previous approaches on the ALFWorld and VirtualHome benchmarks. |
CELLO: Causal Evaluation of Large Vision-Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large vision-language models have improved causal reasoning abilities . however, current models struggle with tasks like causal reasoning . |
| Approach: | They propose a fine-grained and unified definition of causality involving interactions between humans and objects. |
| Outcome: | The proposed model surpasses traditional commonsense causality by including explicit causal graphs . it also shows that current LVLMs can benefit from a causally inspired prompting strategy . |
AGENTVIGIL: Automatic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents (2025.findings-emnlp)
Copied to clipboard
Zhun Wang, Vincent Siu, Zhe Ye, Tianneng Shi, Yuzhou Nie, Xuandong Zhao, Chenguang Wang, Wenbo Guo, Dawn Song
| Challenge: | AGENTVIGIL is a black-box optimization framework to exploit indirect prompt injection vulnerabilities . indirect prompts compromise the core of LLM agents by manipulating contextual information rather than direct user prompts. |
| Approach: | They propose a black-box optimization framework to exploit indirect prompt injection vulnerabilities . they use a Monte Carlo tree-based algorithm to iteratively refine inputs . |
| Outcome: | The proposed framework achieves 71% and 70% success rates against two public benchmarks . |